In a computer system or in a network of computer systems, a common mechanism for storing and sharing units of information/data is through the utilization of a database. A database is a collection of information that has been organized to facilitate accessing, managing and updating. Commonly implemented database types can include, but are not limited to, flat model, network model, distributed model and relational databases.
Many database types can be a combination of instance and data storage in which an instance may consist of a set of operating system processes and memory structures that interact with the storage. PMON (process monitor) and SMON (system monitor) are typical system processes. The memory structure is commonly called the SGA (System Global Area) which typically retains cache information including, but not limited to, user data, data buffers, and SQL (structured query language). In addition to instance and data storage, many databases may include redo logs. Redo logs retain transactional history (e.g., an online redo log) and can be archived into archive logs (e.g., an off line redo log), both of which are critical for the recovery of data.
Included in some database types is a database messaging system that enables propagating, managing and sharing of data and units of information. The propagating, managing and sharing can be within a database or from one database to another database. A unit of information, also referred to as an event, may be a database change or an application generated message. The messaging system, or event notification system provides a unified framework for processing events. Events generated in applications, in workflow or implicitly captured from redo logs or database triggers can be captured in a queue. These events can be consumed in a variety of ways. Events can be automatically applied with a user-defined function or database table operation, can be explicitly de-queued, or a notification can be sent to the consuming application. If the consuming application is located on another database, then the messaging system automatically propagates the events to the appropriate database.
Publishers/subscribers to the database can share these events through utilization of a messaging system. The publisher/subscriber can control what events are put into the messaging system, how the messaging system flows or is routed from database to database, what happens to messages in the messaging system as the messages flow into each database and how the messaging system terminates.
Predicated upon publisher/subscriber specifications, a messaging system may capture, stage and manage messages in the database automatically including, but not limited to, data manipulation language (DML) changes and data definition language (DDL) changes. A publisher/subscriber can also enter user-defined messages in the messaging system, and the messaging system can propagate the messages to other databases or applications automatically. When messages reach a destination database, the messaging system can consume them based on publisher/subscriber specifications.
In such a publish/subscribe environment, there is a constant inflow of published events which are then consumed by the subscribers. These events may be transformed into other events depending upon the subscriber's expressed intent. The lifetime of an event may be described as event creation, event consumption and, optionally, event transformation. The shape and size of the events are dependent upon the data that is flowing through the messaging system.
Database changes are logged persistently to redo log files. It is noted that in many databases, the redo log can be quite substantial in size and volume. These redo log files are continually mined to extract new changes, construct a corresponding event for the change and then publish the event in a database queue for interested subscribers. Additionally, the number or volume of events in the redo log, from which the messaging system retrieves an event, may cause the messaging system to utilize substantial system resources in the publication and consumption of these events.
In a high performance environment in which a messaging system is implemented, the critical stages of event processing, e.g., creation, consumption and optionally transformation, need to be optimal. In particular, when event throughput is varied and high volume (thousands of events per second), the critical stages of event processing can consume substantial system resources including, but not limited to, processor usage, memory usage and bandwidth. Further, in a high performance environment that deals with thousands of events per second, the creation, transformation and consumption of events can be a significant portion of the lifetime of the event. Specifically, creation and marshaling (e.g., transformation) may cause the bulk of the event processing time. These critical stages can adversely affect the scalability of the messaging system.
Further, events can be arbitrarily complex and they can be nested such that an event may be comprised of sub-events, within sub-events, within the event, e.g., having a hierarchical form. An event can also have a subdivided form including sub-events that can include a collection of sub-events, taking a large hierarchical form. During an event creation process, the messaging system examines the entire structure of the event, creates the event each time a value is to be set for that event, and performs a memory allocation for that event. During an event destroy process, the messaging system has to examine the entire memory allocation for that event.
When there is a constant allocation and de-allocation of small pieces of RAM, it can lead to memory fragmentation. This may prevent the database messaging system from allocating contiguous blocks of RAM. Memory fragmentation can cause, but is not limited to, elevated central processing unit (CPU) usage and increased retrieval time leading to higher latency in marshaling events. This can cause a chain reaction in other modules and introduce significant latencies into the entire event processing framework.
Thus, many current databases and associated messaging systems may not provide an efficient mechanism for event storage and/or retrieval of the events. Accordingly, there exists a need for a high-performance scalable, adaptable multi-dimensional event repository for utilization in conjunction with a database messaging system's storing and retrieving of events.